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(54) Automatic content-analysis based use of colour correction process 



(57) The present invention is directed to a method 
for colour correction for images, in particular for digital 
still camera images, including the following steps: an im- 
age is analysed with respect to particular features as- 
signed to a variety of image classes, each class having 
particular characteristics; the analysis based on a com- 



parison of the image data of said image with a data base 
including data for assigning said image to a particular 
image class; said particular image class is associated 
with at least one colour correction process adapted to 
implement colour correction for the assigned image 
class. 
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Description 

[0001] The present invention is directedto a method for colour correction of images according to claim 1 . Furthermore, 
the present invention is directedto a data carrier for the method according to claim 1 . In addition, the present invention 
is directed to an electromagnetic carrier wave carrying control data in accordance with the method according to claim 1 . 
[0002] In the prior art, many attempts have concentrated on the correction of digital colour data belonging to images 
of any kind. In particular, scanned negative or positive films have been transformed to digital data and have been 
directed to several different processes or methods to correct the colours of these image colour data. In addition, many 
pictures taken by consumers by means of digital still cameras also suffer from colour deviations which can also be 
based on the application of particular colour corrections accomplished by the software of the cameras themselves. It 
is known that photofinishing laboratories printing photos on the basis of the digital image data taken by still cameras 
have to manually correct the colours of up to 50 % of the images before printing out these images. Otherwise, the 
colour quality of these prints would not be deemed to be sufficient by the customer. Accordingly, the cost of printing 
out in particular images taken by digital still cameras incurs considerably higher costs because the colour correction 
has to be accomplished manually. 

[0003] Many colour correction algorithms are available. However, each of these colour correction algorithms or proc- 
esses only provides satisfactory results for a very limited range of images. Furthermore, the known colour correction 
processes are based on the assumption that calibrated systems are used to take the photographs, whereas, digital 
still cameras, however, already implement an automatic correction or adaptation of the colour data of an image. Thus, 
since typical digital still cameras already transform colour image data into a particular colour space in the processor 
of the digital still camera, the known colour correction processes are not able to work with colour image data recorded 
by typical digital still cameras. 

[0004] As outlined above, each of the usual colour correction processes is only applicable with acceptable results 
to certain types of images with certain features. On the other hand, the same colour correction processes render bad 
results for other types of images having other characteristics. Therefore, it is not possible to use the known colour 
correction processes for all types of images. Furthermore, as far as manual colour correction is concerned, it is not 
possible to implement this without the interaction of an educated and experienced, and thus very expensive operator. 
[0005] Accordingly, it is an object of the present invention to develop a method for colour correction for images, and 
in particular for digital still camera images, which can resolve out the shortcomings of the known prior art and in particular 
of the above-described prior art. In particular, it is an object of the present invention to provide a new method able to 
implement a colour correction for a variety of different types or kinds of images, without the interaction of an operator. 
[0006] At least one of the above objects is solved by a method in accordance with claim 1 . A data carrier in accordance 
with claim 1 0, as well as an electromagnetic carrier wave in accordance with claim 1 1 , also considerably improve the 
known prior art in the same way 

[0007] The advantages of the present invention are based on a method which is, however, particularly useful for 
images taken by digital still cameras, and in general includes the following steps: An image is analysed with respect 
to particular features assigned to a variety of image classes, each class having particular characteristics; then an 
analysis being based on a comparison of the image data of said image with a data base including data to assign said 
image to a particular image class is carried out; and said particular image class is associated with at least one colour 
correction process which is adapted to implement a colour correction for the assigned image class, and thus for said 
image which has been identified as belonging to that particular class. Since many simple but effective colour correction 
algorithms or processes are specialised to particular colour defects and cannot deal with other colour defects, it has 
been necessary to employ the help of an experienced photofinishing operator to correct defective images taken by 
digital still cameras. This has been time consuming, and also therefore involved high costs. In accordance with the 
invention, it is not possible to automatically decide which colour correction process is promising with respect to which 
image, since it is possible to identify whether this image includes particular semantic or generic features, or other 
features to identify a particular images as belonging to a particular image class, which can be corrected by one or 
several simple but specialised and very effective colour correction algorithms. 

[0008] Therefore, in accordance with the present invention it is possible to automatically decide by means of a hier- 
archical scheme which image has to beprocessed by means of which colour correction algorithm orprocess. Therefore, 
well known colour correction algorithms like white patch, grey world or retinex, and other well known colour correction 
processes, can be automatically directed to photographic images and in particular to the digital data of images taken 
by digital still cameras. 

[0009] In other words, the present invention is directed to a method for colour correction which uses a hierarchical 
scheme. According to this scheme, an image is first analysed with respect to particular features. If one or more of these 
features are determined to be present in the image, it is determined whether the particular image belongs to a certain 
image class, said class always including at least one or more of the features identified as being present in the particular 
image. Then, one or more of the colour correction processes assigned to the one or more image classes to which the 
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particular image belongs can be applied to the particular image to correct the colours of the image. 
[0010] Of course, once the particular image class and thus the particular one or more colour correction processes 
have been determined, it is possible to choose one of a number of colour correction processes deemed to be useful 
for the selected image class. It is also possible to apply a number of the colour correction processes in a particular 

5 sequence to correct the colours of the particular image. 

[0011] The decision as to which of the selected colour correction processes will work with the colour data of the 
particular image to be corrected can depend on probabilities or parameters. For example, if it is very likely that a 
particular colour correction algorithm can be applied to a particular image and another colour correction algorithm or 
process has a very small probability of being applicable, it might be advisable to first use the colour correction process 

w with the smallest probability with small parameters to impose small corrections, and afterwards to apply the colour 
correction process with the large likelihood and with large correction parameters, resulting in large alterations of the 
colours. This is deemed to be useful since the colour situation is not greatly changed by the colour correction algorithm 
with the large probability and large alteration parameters if the selected colour correction process with the small prob- 
ability and the small alteration parameters, and therefore the small alteration results, is applied first. However, this 

15 situation may be completely different for particular images in particular situations. 

[0012] Of course, it is also advantageous to apply the colour correction process which is most likely for a particular 
image belonging to a particular selected image class. In some instances, it might be sensible to direct a further image 
class analysis to the already processed image data of said particular image to determine whether the image class has 
been changed during processing and whether it is then reasonable to direct a further, possibly different colour correction 

20 algorithm or process to the same image, to deal with other colour defects, after a further image class analysis. 

[0013] A further variation of the method of the present invention involves a probability threshold which is used to 
select one or more colour correction algorithms or processes from a variety of colour correction processes deemed to 
be useful for a particular image. 

[0014] According to a further useful variation of the present invention, further image information can be taken into 
25 account to pre-process the digital data of an image or to adapt the one or more selected colour correction processes. 
Accordingly, it is possible to read additional data from the memory of a digital still camera, such as the day, time, the 
type of camera, the type of correction and recording algorithms used in that particular digital still camera, whether a 
flash was used, and so on, to obtain additional information on how to correct the colours. 

[0015] It is a very important further aspect of the invention to apply a Bayesian data analysis, and accordingly the 
30 Bayesian inference, to the hierarchical scheme of this invention. With respect to this aspect, reference is made to: D. 
Sivia. Data Analysis — A Bayesian Tutorial. Clarendon Press, Oxford, 1 996, which reference is incorporated into the 
present disclosure. 

[001 6] On the basis of the Bayesian inference, it is additionally or alternatively possible to combine any kind of colour 
correction process, whether it is currently known or not, in order to achieve an acceptable colour correction. 

35 [0017] According to a further aspect of the present invention, images which cannot be identified as belonging to a 
particular image class can be directed to the attention of an operator, or it is possible to store the image data of such 
images to be processed in a batch for later, so that it is usually possible to proceed automatically in a time and cost 
saving manner. Only if images are deemed not to be colour correctable by means of any of the available colour cor- 
rection processes, are the image data of these uncorrectable images stored, to be transmitted to an operator for colour 

-*o correction, or are directly transmitted to the operator for colour correction. 

[0018] As outlined above, the present invention is also directed to a physical data carrier which carries the control 
data and/or the data base with the features of the different image classes. It is possible to read the control data and 
the data base from this physical data carrier in order to be able to operate a computer or a printer with a computer in 
accordance with the method according to the present invention. 

45 [0019] In addition, the invention is also directed to an electromagnetic carrier wave which supports the control data 
and/or the data base for operating a computer or a printer with a computer in accordance with the method as set forth 
above. 

[0020] In the following, ah advantageous embodiment with features of the invention will be described. Several var- 
iations are also disclosed. The embodiment discussed below is clarified with reference to the drawings, in which: 

50 

Fig. 1 is a rough block diagram of a method with features of the invention; 

Fig. 2 is a variant or more detailed version of a part of the diagram of Fig. 1 ; 

Fig. 3 is a more detailed representation of an image classification as a block diagram; 

Figs. 4 and 5 are distributions for digital cameras in general (Fig. 4) and for a particular camera (Kodak DC 210 
55 Zoom) (Fig. 5). 

[0021] First, the invention will be described in general without reference to the accompanying Figures. 

[0022] In accordance with the invention, it is possible to use the known principals of content-based image retrieval 
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(CBIR; for a recent overview of the research, see A. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content- 
based image retrieval at the end of the early years. IEEE PAMI, 22(12): 1349-1380, December 2000, the content of 
which is included in this application by reference). Accordingly, a large database of images can be accessed by content, 
in most cases by providing an example image and receiving feedback from the subsequent search results. 
[0023] In order to answer content-based "query-by-example" queries, a generic signal-oriented content index has 
to be set-up and compared with the user examples. It is possible to simply describe the images using low-level features 
such as colour, texture, and shape resulting in a large low-level feature vector, which can be used to calculate similar- 
ities. However, these straightforward approaches were not able to demonstrate the ability to express content on a 
higher abstraction level. 

[0024] In accordance with the invention, a more sophisticated hierarchical scheme can also be used. In a first aspect, 
it is possible to apply a hierarchical scheme that first determines a "codebook" or set of characteristic classes based 
on low-level features, for instance using self-organising maps or vector quantisation. These classes can then be linked 
to semantically meaningful labels. 

[0025] Most of the simple colour correction algorithms (white patch/grayworld/retinex) perform well on some images 
and badly on others. Therefore, if it is possible to automatically detect the images which can be successfully treated, 
it will be possible to apply the simple algorithms in an optimum way: for each characteristic image class, the optimum 
parameters of each algorithm can be selected. It is possible to determine which algorithm or which combination of 
algorithms performs best. 

[0026] In order to obtain the characteristic image classes, a modified version of the hierarchical scheme presented 
in M. Schroder, H. Rehrauer, K. Seidel, and M. Datcu "Interactive learning and probabilistic retrieval in remote sensing 
image archives", IEEE Trans, on Geoscience and Remote Sensing, 2000 (submitted for publication) is used. The 
disclosure of this reference is incorporated in the present application by reference. According to this modification, the 
image content description is performed in a hierarchical manner as depicted in Table 1 , and in more detail in the scheme 
of Fig. 2. At first, local image features are extracted by means of various models 21 (e.g., colour, density, texture). 
Then , in a first classification step, the local features are classified into characteristic local classes 22 from which features 
in a block 23 can be calculated of the whole image (e.g., colour complexity, density complexity). Combined with addi- 
tional prior knowledge from the metadata of the original image (e.g., camera model, exposure settings), the image can 
be classified into a characteristic image class 24. 

[0027] The modification can also be commented on with reference to the following table 1 : 

Table 1 : 



Hierarchical scheme of generic content description. 


Level 


Explanation 


Example 


5 


Semantic image class A^ 


class "white patch performs poorly" 


4 


Generic image class co s 


class "high colour complexity" 


3 


Image feature 


colour complexity 


2 


Pixel class 




1 


Pixel feature 


colour, texture 


0 


Image data l k 





[0028] For a certain image l k , of which automatic colour correction is applied, the probabilities 



P(wi|4) 



(1) 



50 



are obtained, that is, the probabilities of the various image classes u>j for a given image / k . 
[0029] Similarly, in a supervised scenario, 



55 



p(Aa)l/ k ). 



(2) 



[0030] If enough manual colour correction ground truth is available, a model can be set-up of how well a certain 
algorithm (e.g. "greyworld" with certain parameter settings) is able to reproduce it. 
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[0031] Mathematically, the class dependent model for colour correction can beexpressed as 

P(eia>i,/ k ) ( (3) 

5 

wherein: A given image (/ k ); a certain image class (oi); and a certain colour transformation (6), e.g., if we linearly scale 
each rgb channel, then we will have 6 = (r f ,g f .b f ). 

[0032] Given a particular image l k , we are interested in determining the probability distribution p(6ll k ) of the colour 
transformation 6. From this distribution, the transformation to be applied can be obtained by maximisation. 
10 [0033] The probability distribution p(0ll k ) can be expressed in terms of the image classes {o)j} as 

P(0|ik) =Ep(0Mk)-P(<*,Hk), (4) 

i 

15 

where the sum over i extends over all image classes coj. The individual factors of the elements of this sum are obtained 
as described above. In summary, the posterior image class probability p(u> jf l k ) can be determined by the classification 
as outlined above. The distribution of colour correction transformation p(Glu>j.i k ) can be derived from training data from 
manual colour corrections, as described in detail previously. 
20 [0034] In summary, it is proposed to precede the application of the one or more simple colour correction algorithms 
by a step involving classification of the image into certain classes and subsequently applying colour correction algo- 
rithms or processes that have been optimised forthe specific image class. These image classes can be either semantic 
classes (certain sujets, e.g., "indoor scene", vegetation scene", or "mountain scene") or signal-oriented, generic classes 
(e.g., "scene of high colour complexity"). After the class-specific application of a set of simple colour correction algo- 
us rithms, the results of these algorithms are combined in such a way as to take into account the class-specific reliabilities 
of each algorithm or process. This can be done in a step involving Bayesian inference that also allows the taking into 
account of further available information (e.g. the camera model or meta data of the exposure). Altogether, this results 
in a hierarchical system of automatic colour correction that significantly reduces the number of consumer images re- 
quiring manual colour correction. 
30 [0035] After a set of colour corrections has been selected, a Bayesian fusion of the results takes place. For a given 
class u>„ a set of automatic colour correction algorithms {Vj} is applied, each of which provides a certain colour correction 
transformation 6 V j(l k ,u)j) (the notation AI B means "A for a given B"). Since for each algorithm/class combination the 
accuracy of the colour transformation (compared to humans determining the transformation "manually" via visual in- 
spection) can be determined, it is possible to model the relationship between the actual colour transformation 6 and 
35 the result of each automatic colour correction as a probability: 

P( GvjOfcUi) | 0 ), (5) 
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50 
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result of algorithm actual 



which is given as, for example a multivariate Gaussian. In this stage, it is the actual colour transformation which has 
to be determined. 

45 [0036] In order to combine the results of several algorithms, the total probability of ail results can be used, which is 
given as a product of the individual probabilities, if we assume the independence of individual algorithms. The prior 
distribution of the colour transformation reads as follows: 



p(Glo)j, meta data) (6) 
[0037] On the basis thereof, the posterior distribution of the colour transformation reads as follows: 

p(0|wi,lk) oc p(0|o) i5 meta data) . np(0 V3 (Ik, (*)|0), (7) 

j 
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where the sum over j extends over all algorithms Vj (the symbol « denotes "proportional to"). Now, it is possible to 
obtain the final colour transformation (for this image class) by maximising Eq. (7). The prior p(eio>j, meta data) is inde- 
pendent of the currently analysed image l k and models the expected colour corrections for the current image class o>j 
and available meta data. 

5 [0038] The invention is structured as follows. First the image analysis is conducted, providing a signal-oriented de- 
scription of the content of the image. After the determination of the class-specific colour corrections, the different results 
are processed in a Bayesian framework, to provide a colour corrected image data file which can be printed out. 
[0039] The hierarchical system of the invention can be applied to any devices in which automatic colour correction 
of consumer images, and particularly images taken by digital still cameras, is needed, such as digital photofinishing 

10 systems (e.g. all the digital systems of Gretag Imaging, Photo CD systems, and stand-alone or independent image 
enhanced tools. 

[0040] The basic workflow according to a preferred example of the present invention will now be explained with 
reference to Fig. 1 . First, the content of the image is analysed using methods of CBI R, as explained above. As a result, 
posterior probabilities 
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PW'k) (8) 

for image classes are obtained. In addition to the unsupervised signal-oriented classes co,-, it is the concept of this 
invention to also allow the use of supervised ("semantic") classes Ay, that might correspond to certain photographic 
sujets. Accordingly, the result is classification of 

P(A v N k ). (9) 

[0041] For the later Bayesian inference process, all a>, simply have to be substituted by \. 

[0042] After classification, a set of automatic colour correction algorithms is applied and the individual results are 

fused into one final result p(a>jl / k ), as explained above. 

[0043] Finally, the combined colour correction results for each class p(o>jl / k ) are combined using the probabilities of 
each class p((0jl / k ) in order to obtain p(6l/ k ). This is the probability distribution of the colour correction for the image 
under investigation. The final colour correction is obtained by maximising this distribution, this final step not being 
mentioned in Fig. 1 . 

[0044] Furthermore, the basic workflow of the present invention first involves a classification 2, followed by a class- 
dependent application of colour correction algorithms 3, and then the Bayesian fusion of the results 4. 
[0045] In addition, Fig. 2 shows the hierarchical image content description. First, the local image features are ex- 
tracted using various models 21 (e.g., colour, density, texture). In a first classification step 22, the local features are 
classified into local classes of characteristics from which the features 23 of the whole image (e.g. colour complexity, 
density complexity) can be calculated. Combined with additional prior knowledge from the meta data of the original 
image (e.g. camera model, exposure settings) the image can be classified into a characteristic image class 24. 
[0046] In Fig. 3, a class dependent application of a set of colour correction algorithms 32 is followed by a Bayesian 
fusion 34 of the set of results 33. As a result, a final correction 8 is obtained at the reference number 35. The number 
of colour correction algorithms is not limited to three. 

[0047] In Fig. 4, prior distributions p(log(r f ), log(g f )) for digital cameras in general (top) and for one particular model 
(Kodak DC 21 0 Zoom, bottom) are depicted for comparison purposes. Accordingly, it can be seen that colour corrections 
for images taken with a digital still camera have to be adapted in accordance with the principles of the present invention . 

Claims 

1 . Method for colour correction for images, in particular for digital still camera images, including the following steps: 

a) an image is analysed with respect to particular features assigned to a variety of image classes, each class 
having particular characteristics; 

b) the analysis is based on the comparison of the image data of said image with a data base including data 
for assigning said image to a particular image class; 

c) said particular image class is associated with at least one colour correction process being adapted to im- 
plement colour correction for the assigned image class. 
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2. Method according to claim 1 , wherein one or more probabilities likelihood's are determined that a particular image 
belongs to one or more classes. 

3. Method according to claim 2, wherein the most likely class is selected and a colour correction process associated 
with said selected class is applied to said image or digital image data thereof. 

4. Method according to claim 2, wherein if several classes are likely to belong to said image, the colour correction 
processes belonging to said classes are applied to said image, if the corresponding probabilities exceed a particular 
probability threshold. 

5. Method according to claim 2, wherein one or more colour correction algorithms or processes are selected and/or 
combined according to the principals of a Bayesian inference. 

6. Method according to one of claims 2 or 5, wherein the colour correction processes are applied either with increasing 
15 or with decreasing probabilities. 

7. Method according to one of claims 1 to 6. wherein further imaging information, such as recording information of a 
specific camera or the like, are taken into account. 

20 8. Method according to one of claims 1 to 7, wherein one or more of the known colour correction processes are used 
to implement said colour correction or colour corrections in accordance with the assignment procedure correspond- 
ing to step 1b) and/or 1c). 

9. Method according to one of claims 1 to 8, wherein images identified as not being colour correctable by means of 
25 any of the colour correction processes are transmitted to an operator for colour correction. 

10. Data carrier, carrying control data and/or analysis data to be used in a computer means, wherein said data imple- 
ment the method according to one of claims 1 to 9. 

30 11. Electromagnetic carrier wave supporting control data and/or analysis data for operating a computer in accordance 
with the method according to one of claims 1 to 9. 
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